Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads

摘要

Within the past few years, organizations in diverse industries have adoptedMapReduce-based systems for large-scale data processing. Along with these newusers, important new workloads have emerged which feature many small, short,and increasingly interactive jobs in addition to the large, long-running batchjobs for which MapReduce was originally designed. As interactive, large-scalequery processing is a strength of the RDBMS community, it is important thatlessons from that field be carried over and applied where possible in this newdomain. However, these new workloads have not yet been described in theliterature. We fill this gap with an empirical analysis of MapReduce tracesfrom six separate business-critical deployments inside Facebook and at Clouderacustomers in e-commerce, telecommunications, media, and retail. Our keycontribution is a characterization of new MapReduce workloads which are drivenin part by interactive analysis, and which make heavy use of query-likeprogramming frameworks on top of MapReduce. These workloads display diversebehaviors which invalidate prior assumptions about MapReduce such as uniformdata access, regular diurnal patterns, and prevalence of large jobs. Asecondary contribution is a first step towards creating a TPC-like dataprocessing benchmark for MapReduce.